Design and quality control of large-scale two-sample Mendelian randomization studies

Abstract Background Mendelian randomization (MR) studies are susceptible to metadata errors (e.g. incorrect specification of the effect allele column) and other analytical issues that can introduce substantial bias into analyses. We developed a quality control (QC) pipeline for the Fatty Acids in Cancer Mendelian Randomization Collaboration (FAMRC) that can be used to identify and correct for such errors. Methods We collated summary association statistics from fatty acid and cancer genome-wide association studies (GWAS) and subjected the collated data to a comprehensive QC pipeline. We identified metadata errors through comparison of study-specific statistics to external reference data sets (the National Human Genome Research Institute-European Bioinformatics Institute GWAS catalogue and 1000 genome super populations) and other analytical issues through comparison of reported to expected genetic effect sizes. Comparisons were based on three sets of genetic variants: (i) GWAS hits for fatty acids, (ii) GWAS hits for cancer and (iii) a 1000 genomes reference set. Results We collated summary data from 6 fatty acid and 54 cancer GWAS. Metadata errors and analytical issues with the potential to introduce substantial bias were identified in seven studies (11.6%). After resolving metadata errors and analytical issues, we created a data set of 219 842 genetic associations with 90 cancer types, generated in analyses of 566 665 cancer cases and 1 622 374 controls. Conclusions In this large MR collaboration, 11.6% of included studies were affected by a substantial metadata error or analytical issue. By increasing the integrity of collated summary data prior to their analysis, our protocol can be used to increase the reliability of downstream MR analyses. Our pipeline is available to other researchers via the CheckSumStats package (https://github.com/MRCIEU/CheckSumStats).

additional, often overlooked source of bias in MR studies are errors in the underlying summary data or metadata.For example, incorrect specification of the effect allele column may lead to effect estimates that are in the wrong direction. 4These errors occur because conventions for the inclusion or naming of data fields that avoid ambiguity have not been widely adopted by the GWAS community, increasing the potential for misinterpretation by data analysts. 5GWAS summary data can also be obtained from an increasingly diverse range of sources, including online platforms and study-specific websites, but it is not always clear whether such results have been through post-GWAS filtering steps [e.g. with low frequency or poorly imputed single-nucleotide polymorphisms (SNPs) excluded], which increases the potential for unreliable genetic associations.The potential for metadata and summary-data errors is compounded in relatively complex MR study designs, such as in MR-PheWAS [6][7][8] (MR-phenome-wide association study), wide-angled MR 7,9 and pan-disease MR, 10 in which summary-data sets from many different studies, corresponding to many different exposures and/or outcomes, are collated and harmonized into a single analysis.Within the GWAS field, quality control (QC) procedures have been developed that can detect a wide range of analytical issues and metadata errors, either at the GWAS stage 11 or at the post-GWAS meta-analysis stage. 12For example, it is common practice to exclude genetic variants of low genotype or imputation quality or with low minor allele counts, since inclusion of such variants can lead to unstable genetic effect estimates and increase the rate of type I errors. 12A widely used QC strategy for the identification of metadata and summary-data errors in GWAS meta-analyses is to compare study-specific statistics to external reference data sets or to results based on theoretical expectations. 12Some of these QC procedures can also be used in the MR context to identify potential issues with the summary data.For example, effect allele coding errors can be identified by comparing reported allele frequency with allele frequency in a reference population.However, MR studies are subject to a unique set of challenges that often hamper the application of some previously developed QC checks.For example, to reduce the risk of individual re-identification, some consortia do not share allele frequency information with external researchers or replace it with the allele frequency of a reference population.A further hindrance is that metrics of genotype or imputation quality, or of between-study heterogeneity (in the meta-analysis context), are often not made available in GWAS results files.Some QC procedures flag potential issues by comparing study-specific statistics across studies 12 but under the assumption that all studies employed the same regression models with the same outcomes, covariates and trait transformations, which is unlikely in complex MR study designs.Some studies only make small subsets of GWAS summary data available to researchers, which makes detecting errors harder.
In the present paper, we describe a pipeline for the QC of GWAS summary data developed for the Fatty Acids in Cancer Mendelian Randomization Collaboration (FAMRC)-a pan-cancer MR study that seeks to evaluate the causal relevance of fatty acids for risk for most major Box 1.General overview of Mendelian randomization studies The main aim of MR is to assess the potentially causal nature and direction of associations between exposure and outcome traits.Rather than studying the exposure-outcome relationship directly, i.e. using phenotypically measured levels of the exposure, MR uses genetic polymorphisms as instruments or proxies for the exposure.If the genetic instrument for the exposure is associated with the outcome of interest, this can be taken as evidence for a causal effect of the exposure on the outcome, so long as instrumental variable (IV) assumptions are met: (i) the instrument is associated with the exposure; (ii) the instrument is not associated with confounders of the exposure-outcome association; and (iii) the instrument is associated with the outcome exclusively through its effect on the exposure.Although violations of assumptions can be introduced by genomic confounding or horizontal pleiotropy, an increasingly sophisticated range of sensitivity analyses are available that can be used to model the impact of such bias on MR findings.
In the two-sample approach to MR, genetic summary data for the exposure and the outcome are obtained from separate studies.This greatly increases the scope of MR, as it means the method can be applied to any disease case-control collection regardless of whether the exposure has been directly measured or not.The success of GWAS has greatly increased the number of traits with available genetic association or summary data.In principle, any heritable trait with summary genetic association data can be used to define an exposure or an outcome in a two-sample MR study and thus the scope for what counts as an exposure or an outcome is very broad.Exposure and outcome traits can vary from relatively simple molecular traits, such as expression or protein traits, to highly complex traits, such as human behaviours and disease outcomes.
cancers.The basic principle of our QC approach is to identify metadata errors through comparison of studyspecific statistics to external reference data sets [e.g. the National Human Genome Research Institute-European Bioinformatics Institute (NHGRI-EBI) GWAS catalogue and 1000 genome super populations] and to identify potential analytical issues or summary-data errors through comparison of reported to expected genetic effect sizes.Using the pipeline, we created a data set of 219 842 genetic associations with 90 cancer types, generated in analyses of 566 665 cancer cases and 1 622 374 controls in 51 studies.The size and complexity of the FAMRC make it an ideal collaboration in which to develop and evaluate QC processes for the detection of errors that can introduce biases into downstream MR analyses.

Methods
The FAMRC had four key design components: (i) fatty acid instrument selection strategy; (ii) cancer outcome selection strategy; (iii) cancer data preparation and harmonization; and (iv) identification of summary-data errors, metadata errors and other analytical issues (Figure 1).

Fatty acid instrument selection strategy
We searched for GWAS of fatty acids published up to December 2018 by searching the NHGRI-EBI GWAS catalogue 13 (https://www.ebi.ac.uk/gwas/) and Open GWAS 14 (https://gwas.mrcieu.ac.uk/), using fatty acid-related search terms, including: fat, acid, fatty acid, DHA, omega, monounsaturated, monounsaturated, polyunsaturated, saturated, omega 3 and omega 6. Fifteen studies were identified by this strategy.When full summary association statistics were available, independent genetic associations with P < 5e-8 were identified through linkage disequilibrium (LD) clumping (r 2 threshold set to 0.01), with LD reference panels based on either the European or East Asian 1000 genome superpopulations (clumping was performed using the ieugwasr package 15 ).We also selected all SNP associations reported in the GWAS catalogue, with no specified P-value threshold.We further identified SNP proxies, defined as SNPs having an r 2 of !0.8 with any one of the fatty acid SNPs in European or East Asian 1000 genomes reference data.We also searched for alias reference SNP identifiers (rsids) in the Single Nucleotide Polymorphism Database (dbSNP) and 1000 genomes reference data, to make allowance for different rsids across different genome builds for the same SNP.We refer to the genetic associations for fatty acids, their r 2 proxies and alias rsids as the 'fatty acid SNP set' (Figure 1).To identify metadata errors, summary-data errors or other analytical issues, we developed and applied a QC pipeline to the fatty acid summary-data sets (described below).

Cancer outcome selection strategy
We searched for studies of cancer in the GWAS catalogue 13 up to 1 November 2018.Search terms included: cancer, carcinoma, neoplasm, neoplastic, tumor, tumour, adenocarcinoma, glioblastoma, leukemia, lymphoma, melanoma, meningioma, mesothelioma, myeloma, neuroblastoma and sarcoma.When multiple studies of the same cancer outcome were identified, we prioritized the larger study.When not already available via Open GWAS 14 (https://gwas.mrcieu.ac.uk/) or the GWAS catalogue, we invited the identified studies to share summary data for all SNPs in their GWAS analysis (defined as 'full GWAS summary data').If studies were unable to share full summary data, they were invited to share genetic association results for the 'fatty acid SNP set'.We further downloaded summary association statistics for cancers from Biobank Japan [16][17][18] (http://jenger.riken.jp/en/),FinnGen [data freeze 1 (14 January 2020)] (https://www.finngen.fi/fi)and UK Biobank 19,20 using the Open GWAS platform 14 and ieugwasr package. 15We prioritized studies of cancer incidence and excluded studies of cancer survival, mortality or progression-related phenotypes.
For data sets obtained via correspondence, studies were invited to share summary data up until December 2019, after which data collection was closed.Example data sharing instructions can be found in the Supplementary materials (available as Supplementary data at IJE online).For each SNP, we asked studies to provide a minimum of: effect estimates (log odds ratios and standard errors), the effect allele, non-effect allele and effect allele frequency.We also asked studies to provide metrics of SNP genotype quality, such as P-values for Hardy-Weinberg equilibrium (HWE) and metrics of imputation quality, such as info scores.When the GWAS was a meta-analysis of multiple independent studies, we additionally requested P-values for between-study heterogeneity.

Cancer data preparation and harmonization
For each cancer with full summary data, we extracted the following three sets of SNPs: i. the fatty acid SNP set; ii. the 1000 genomes reference set; iii. the GWAS catalogue 'top hits' for cancer.
'Top hits' refers to the strongest or statistically significant genetic associations for the cancer of interest from published studies (often defined as P-values <5Â10 -8 ).When full summary data were not provided, QC analyses were restricted to the 'fatty acid SNP set'.We next formatted the cancer summary-data sets to have similar tabular formats (e.g.where results were distributed across multiple files we merged these together) and to have consistently named data fields.SNPs without rsids were mapped to an rsid using the reported chromosome and base pair position.We excluded duplicate and triallelic SNPs as well as SNPs with missing effect sizes and standard errors or with a minor allele count of <50 in either cases or controls.If standard errors were not reported, we attempted to infer these from confidence intervals or P-values before excluding the SNP.We asked each study to confirm the identity of the effect allele and effect allele frequency columns in their data sets, unless this was unambiguously specified in the metadata or associated readme file.We manually mapped the cancer name for each data set to the experimental factor ontology (EFO). 21

QC pipeline to identify analytical issues or summary and metadata errors
3][24][25][26][27][28][29][30][31][32] We used the pipeline to: (i) confirm the identity of the effect allele column (Figure 2); (ii) confirm the identity of the effect allele frequency column (Figure 2); and (iii) identify analytical issues or potential errors in the summary data (e.g. an unusual number of GWAS hits or unusual distributions in effect sizes).All the functions and tests of the QC pipeline are available to other researchers via the CheckSumStats package (https://github.com/MRCIEU/CheckSumStats).

Instrument-specific QC
To identify potential analytical issues or errors in the genetic instruments for fatty acids, we compared genetic association results identified through LD clumping (r 2 ¼0.01 and kb ¼ 10 000) to associations in the GWAS catalogue.We set the significance threshold for LD clumping to the threshold reported in the fatty acid GWAS: 5-e8 in CHARGE (Cohorts for Heart and Aging Research in Genomic Epidemiology Consortium), 33 SCHS (the Singapore Chinese Health Study) 34 and NHAPC/MESA-CHI (Nutrition and Health of Aging Population in China/ Multi-Ethnic Study of Atherosclerosis-Chinese ancestry participants), 35 1e-8 in the FHS (the Framingham study), 36 2.3e-9 in Kettunen et al. 37 and 1.03e-10 in the TwinsUK/ KORA (Twins United Kingdom/Cooperative Health Research in the Region of Augsburg) study. 38We used the 1000 genomes European superpopulation as an LD reference panel for CHARGE, the FHS, Kettunen et al. and TwinsUK/KORA, and the East Asian superpopulation as an LD reference panel for the SCHS and NHAPC/MESA.We searched the GWAS catalogue for the lead SNP, identified by the latter clumping procedure, as well as SNPs within 200 000 base pairs of the lead SNP (associations were retrieved from the GWAS catalogue via the gwasrapidd package 32 ).Data sets were flagged for further investigation if any lead SNPs were absent from the GWAS catalogue.We additionally searched for metadata and summary-data errors in the fatty acid GWAS results through comparisons of effect alleles and allele frequency with external reference data sets and by comparing reported to expected effect sizes (described below).

Confirming the effect allele column
To identify incorrect specification of the effect allele column, we compared summary association statistics in the test data set (either a fatty acids or cancer data set) to summary association statistics in the NHGRI-EBI GWAS catalogue 13 (Figure 2).The latter is a manually curated database of 251 401 genetic associations from 4961 publications (as of April 2021) and includes information on effect alleles, effect sizes and EFOs.The genetic associations in the manually curated database typically correspond to the statistically significant findings ('top hits') from published studies (often defined as P < 5e-8).In recent years, the GWAS catalogue has started to host full GWAS summary statistics.However, for this QC step, we are referring exclusively to the manually curated database of published 'top hits'.
In the first step, we searched the GWAS catalogue for SNPs associated with the EFO or reported trait of the test data set.Second, for each SNP associated with the EFO term, we extracted from the GWAS catalogue the effect size, standard error, effect allele, effect allele frequency and study ancestry (genetic associations were retrieved via the gwasrapidd package 32 ).SNPs missing any of this information, or that were palindromic, were removed.Third, genetic associations for these SNPs were then extracted from the test data set.Fourth, the effect sizes and effect allele frequencies from the GWAS catalogue and test data set were harmonized to reflect the same effect allele and compared in scatter plots (constructed using the ggplot package 22 ).Comparisons were restricted to populations of European or East Asian ancestry.
We then inspected the scatter plots for conflicting directions of association.For example, we declared a conflicting direction of association if the effect allele was associated with higher cancer risk in the GWAS catalogue but was associated with lower risk in the test data set.The level of conflict was further labelled as 'high' if the P-value for the association was <0.0001 in both the GWAS catalogue and the test data set and as 'moderate' if not, so as to make allowance for chance deviations in effect direction in small studies.If the test data set and the data in the GWAS catalogue corresponded to the same publication, the conflict level was labelled as 'high' regardless of the P-value strength.For comparisons of allele frequency, we declared a conflict if the effect allele frequency was not greater (or less) than 0.5 in both data sets.The level of conflict was further labelled as high if the minor allele frequency was 0.4 in both data sets, and as moderate if not.The latter step makes allowance for chance deviations in allele frequencies for SNPs with minor allele frequencies close to 0.5.Conflicts were also labelled as high if the allele frequency differed by >10 points between the test and reference data sets.When interpreting the scatter plots, it is important to take into account the total number of SNPs in the comparison as well as the ancestry of the test and reference data sets.Conflicting associations are more likely to reflect true effect allele coding issues when the conflict is systematic across a large number of SNPs and when the ancestry of the data sets being compared is the same.When a substantial proportion of SNPs displayed effect or allele frequency conflicts, we flagged the test data set as containing a potential effect allele metadata error.

Confirming the effect allele frequency column
To confirm the effect allele frequency column, we compare the test data sets to two types of reference data sets: the 1000 genomes project 39 and the exposure study.In the case of the present analysis, we used CHARGE and the SCHS as representative of exposure (i.e.fatty acid) studies in Europeans and East Asians, respectively.For comparisons with the 1000 genomes project, we created a reference data set of 2297 SNPs that have the same minor allele across the African, European, East Asian, American, South Asian and Global super populations and that also have a minor allele frequency of between 0.1 and 0.35 (this data set is available to other researchers in the CheckSumStats R package) (Figure 2).We refer to the 2297 SNPs as the '1000 genomes reference set'.For comparisons with CHARGE and the SCHS, we created a reference data set corresponding to the fatty acid SNP set described above (see 'Fatty acid instrument selection strategy').We then compare minor allele frequencies between the test data set and the reference data set.The comparison involves the following steps.First, we merge the test and reference data set.Second, we recode the reported effect allele and reported effect allele frequency in the test data set to reflect the minor allele in the reference data set.Third, we compare minor allele frequencies between the data sets in scatter plots 22 and inspect the plots for conflicting patterns.A conflict is declared for individual SNPs if their allele frequency is >0.5 in the test data set.If the frequency is also !0.58, the conflict level is upgraded to 'high' (to make allowance for chance deviations).Conflicts are also labelled as high if the allele frequency differs by >10 points between the test and reference data sets.If an inverse correlation is observed across the vast majority of SNPs, this indicates that the conflict is systematic and that the reported effect allele frequency actually corresponds to the non-effect allele.When there is a conflict for approximately half the SNPs, this implies that the reported effect allele frequency column actually corresponds to the minor allele and that the minor allele is not consistently the effect allele.In the latter situation, the scatter plot will show two separate groups of SNPs-one with a positive correlation and the other with an inverse correlation-in the allele frequency between the data sets.The strength and linearity of the correlation in the allele frequency between the test and reference data sets also provide information on the ancestral background of the participants used to generate the test data set.An advantage of using our '1000 genomes reference set' is that incorrect specification of the effect allele frequency can be identified without knowledge of the ancestral background of the test data set.

Identifying other analytical issues and summary-data errors
To identify potential analytical issues or summary-data errors, we compare the expected and reported effect sizes.For continuous exposures, such as fatty acid levels, we generate expected betas using the formula: where p is the minor allele frequency, n is the sample size and z is the ratio of the effect size to its standard error.The predicted effect size from this transformation can be interpreted as the standard deviation change in the exposure per copy of the effect allele (assuming that z itself was generated in an additive genetic model).When the expected effect size is a log odds ratio, e.g. for cancer status analysed in a logistic regression model, we generate the expected log odds ratio for each SNP using a simulation method that takes into account the SNP's z-score, minor allele frequency and the number of cases and controls 40 and assumes an additive genetic model.More details of the method can be found in the Supplementary materials (available as Supplementary data at IJE online).
We then regress the expected effect size (the per-allele standard deviation change in a continuous trait or log odds ratio for a binary trait) on the reported effect size and interpret a slope very different from 1 (which we define as either >1.20 or <0.8) as evidence for an unusual distribution in the reported effect sizes.We also assess the overall shape of the relationship between the expected and reported effect sizes in scatter plots, with the expectation of linearity.Deviations of the slope from one or non-linear patterns could reflect: i. errors in the reported effect sizes, reported sample sizes or reported allele frequencies; ii.effect size scale conflicts [e.g. reported effect sizes have not been standardized (for continuous traits) or effect sizes have not been generated in a logistic regression model (for binary cancer outcomes)]); iii. the impact of covariate adjustment in the regression model; iv.deviations from HWE.
If we found that the summary association statistics for cancer were generated in a linear model (e.g.BOLT-LMM 41 ), we transformed the effect size to a log odds ratio scale using the following formula: where u is the case prevalence in UK Biobank.The standard error for the log odds ratio can be obtained with the same transformation.
To see whether discrepancies between the reported and expected effect sizes were related to metrics of genotype or imputation quality, we compared discrepancies to reported info or r 2 imputation scores, P-values for HWE and, in the case of meta-analyses, P-values for between-study heterogeneity and the number of studies.Potential errors in reported effect sizes were also identified by comparing z bscores (inferred from the reported effect size and standard error) to z p -scores (inferred from the reported P-value) in scatter plots (also known as P-Z plots 12 ).

Fatty acid data sets
[36][37][38][42][43][44][45][46][47][48][49][50] The median number of fatty acids assessed per publication was 6 (minimum ¼ 2, maximum ¼ 34).Nine of the 15 publications were conducted by, or overlapped with, the CHARGE consortium.The median study sample size was 7811 (minimum ¼ 284; maximum ¼ 17 267).The 15 publications corresponded to seven independent studies or consortia.An interaction study was the only fatty acid GWAS excluded. 51We subsequently invited the identified studies to share full summary data with the FAMRC (except for Shin et al. 38 and Kettunen et al., 52 which were already available via Open GWAS 49 ).The vast majority of the GWAS analyses were conducted in European ancestry populations (11/15), two were conducted in populations of East Asian ancestry, one in a population of South Asian ancestry and one in a transethnic GWAS of Europeans and East Asians.We collated full summary data from 13 of 15 publications, corresponding to six independent consortia or cohort studies: CHARGE, 33,42-44,47-50 SCHS, 34 FHS, 36 the TwinsUK/ KORA study, 38 the NHAPC/MESA-CHI study 35,47 and Kettunen et al. 37 In the SCHS, fatty acid GWAS analyses were conducted separately amongst myocardial infarction cases and controls.We combined these data sets by fixedeffects meta-analysis in METAL. 53Overall, 124 summarydata sets were available from the six studies (where each data set corresponds to a single fatty acid GWAS analysis, Supplementary Table S2, available as Supplementary data at IJE online).
To identify metadata and summary-data errors, we applied a custom QC pipeline to the CHARGE, FHS, SCHS, TwinsUK/KORA, NHAPC/MESA-CHI and Kettunen et al. studies (Figure 3 and Supplementary Figures S1-S6, available as Supplementary data at IJE online).No allele frequency or effect allele conflicts were observed, indicating that the reported effect allele and effect allele frequency columns were correctly indicated.A strong and positive linear relationship between the expected and reported effect sizes was also observed in the FHS, TwinsUK/KORA, NHAPC/MESA-CHI and Kettunen et al. studies, with slopes close to 1, suggesting the absence of major analytical issues in these studies.
The expected and reported effect sizes for selected fatty acids were not, however, well correlated in the CHARGE (Figure 3) and SCHS studies (Supplementary Figure S5, available as Supplementary data at IJE online).We also identified 109 independent GWAS hits for arachidonic acid in CHARGE after LD clumping, of which only four were also reported in the GWAS catalogue, compatible with the presence of a large number of false positives.After corresponding with the data provider, we were able to confirm that post-GWAS filtering of low-quality variants (defined as SNPs with minor allele frequency of <5%, imputation r 2 < 0.5 or as SNPs that were present in only one study 33 ) had not been performed on the data set posted to the CHARGE website.After excluding these SNPs, following recommendations of the data provider, 33 we observed a strong linear relationship between the reported and expected effect sizes and a slope of 1.02 (Supplementary Figure S7, available as Supplementary data at IJE online).The 109 independent GWAS hits also decreased to seven in the cleaned data set, all of which mapped to the fatty acid desaturase genomic region on chromosome 11 or Pyridoxal Dependent Decarboxylase Domain Containing 1 (PDXDC1) on chromosome 16, regions harbouring established GWAS hits for fatty acids (and therefore unlikely to be false positives).In the SCHS, the relationship between the expected and reported effect sizes was skewed by a single outlier SNP (Supplementary Figure S5, available as Supplementary data at IJE online).Further investigation revealed that the outlier was due to incorrect specification of the sample size for this SNP.
We also identified two independent GWAS hits for selected fatty acids in the NHAPC/MESA-CHI and SCHS studies that were not present in the GWAS catalogue or in the associated publications.We subsequently confirmed that post-GWAS filtering steps for low-quality variants had not been applied to the GWAS results files for the NHAPC/MESA-CHI study and that the identified GWAS hit had failed the reported QC checks (we therefore excluded this variant).In the SCHS, correspondence with the data provider indicated that a file sharing error had occurred and we therefore obtained a new set of GWAS results files (in which conflicts with the GWAS catalogue were no longer observed).Conflicts with the GWAS catalogue were not observed for the FHS, TwinsUK/KORA and Kettunen et al. studies.][44][47][48][49][50] After applying the SNP selection strategy and resolving the analytical issues flagged by the QC pipeline, we identified 288 SNPs associated with 53 fatty acid traits (median 6 per trait).We identified a further 1841 SNP proxies using the 1000 genomes European super population, 2251 SNP proxies in the 1000 genomes East Asian super population and 197 alias rsids in dbSNP and 1000 genomes reference data.The total number of SNPs associated with fatty acids, their r 2 proxies and alias rsids was 2326 for European Figure 3 Quality control report for genetic summary data from a genome-wide association of arachidonic acid in the Cohorts for Heart and Aging Research in Genomic Epidemiology Consortium (CHARGE).Allele frequencies are expected to be <0.50.A high allele frequency conflict is defined as an allele frequency of >0.58 in the test data set (CHARGE in this example) or if the allele frequency differs by >10 points between the test and reference data sets.Moderate allele frequency conflicts are allele frequencies of >0.50 but 0.58.Effect size conflicts are defined as different directions of effect, represented by signed z-scores, between the test data set (CHARGE in this example) and the GWAS catalogue.The level of conflict is further labelled as 'high' if the P-value for the association is <0.0001 in both the GWAS catalogue and the test data set, and as 'moderate' if not.Effect allele studies and 2596 in East Asians (excluding duplicate SNPs that overlapped amongst the fatty acid and proxy sets) (Supplementary Tables S3 and S4, available as Supplementary data at IJE online).We henceforth refer to these SNPs as the 'fatty acid SNP set'.

Collation of cancer data sets
As of January 2020, we had collated 166 summary genetic data sets from 54 cancer studies 16, (Supplementary Table S5 and Supplementary Figure S8, available as Supplementary data at IJE online). Thirtyeight studies supplied a single data set, 10 studies supplied two to five data sets and 6 studies supplied more than five data sets.Of the 166 cancer data sets, 31 were from UK Biobank, 12 were from Biobank Japan and 29 were from FinnGen.Fifty-nine data sets (from 41 studies) were obtained via correspondence with study principal investigators, 6 data sets (from 3 studies) were downloaded from the GWAS catalogue and 101 data sets (from 12 studies) were downloaded from the Open GWAS project.14 Of the 101 Open GWAS data sets, 12 were from Biobank Japan, 29 were from FinnGen, 30 were from UK Biobank, 27 were from GWAS meta-analysis consortia and 3 were from other studies.Further details of the cancer studies can be found in Table 1 and Supplementary Tables S5 and S6 (available as Supplementary data at IJE online).Effect allele frequency was available in 156 data sets (from 46 studies), metrics of imputation quality (r 2 or info scores) were available in 53 data sets (from 25 studies) and P-values for deviations from HWE were available in 6 (from 6 studies).Of 65 data sets derived from 29 GWAS meta-analyses, P-values for between-study heterogeneity were available in 18 (from 9 meta-analyses) and the number of studies per SNP was available in 16 (from 7 meta-analyses).
We extracted three sets of genetic associations from each data set for which full GWAS results were available (132 data sets from 31 studies): (i) the fatty acid SNP set, (ii) the 1000 genomes reference set and (iii) known cancer hits in the GWAS catalogue.For 34 data sets from 25 studies, only a subset of GWAS results, corresponding to the fatty acid SNP set, was available.We excluded duplicate and triallelic SNPs; SNPs with missing effect allele, effect sizes or standard errors; SNPs that could not be mapped to an rsid; and SNPs with a minor allele count in cases of <50.After these exclusions, there were 401 026 genetic associations with cancer across 163 data sets in 52 studies.Of these, 223 970 genetic associations corresponded to the fatty acid SNP set, 93 121 corresponded to the 1000 genomes reference set and 24 860 corresponded to known cancer associations in the GWAS catalogue.Three studies providing genetic associations for the fatty acid SNP set provided an additional 40 582 genetic associations for SNPs within 500 kb of a fatty acid index SNP [ACCC (ID3), UCSF_AGS þ SFAGS (ID133) and UCSF_Mayo (ID 134)] (cancer study abbreviations explained in Supplementary Table S6, available as Supplementary data at IJE online).

Results of QC pipeline applied to the cancer data sets
Metadata errors or analytical issues were identified in 41 cancer data sets from 20 studies (Supplementary Table S7, available as Supplementary data at IJE online).These included serious metadata errors (defined as incorrect labelling of the effect allele or effect allele frequency columns) in five data sets from five studies.In three data sets, alleles associated with higher cancer risk in the GWAS catalogue were associated with lower risk in the test data set for a substantial proportion of SNPs (Figure 4 and Supplementary Figures S9 and S10, available as Supplementary data at IJE online), suggesting that the effect allele column refers to the non-effect allele.This was clearest for GliomaScan (ID 967) where 19/21 SNPs were discordant with the GWAS catalogue but was less clear for the NBS (ID 106) and BC-NHL (ID 5) data sets (cancer study abbreviations explained in Supplementary Table S6, available as Supplementary data at IJE online).In the BC-NHL, although all SNPs were discordant to the GWAS catalogue, the number available for comparison was small and z-scores were not large (z 2.5).Therefore, we could not rule out chance deviations from the GWAS catalogue for this data set.In addition, the ancestry of the BC-NHL (East Asian) was different to the ancestry of the reference data set (European).Therefore, the observed conflict for the BC-NHL data set could also reflect differences in LD between populations.In the NBS (ID 106), equal numbers of SNPs were highly discordant and highly concordant to the GWAS catalogue.Due to the ambiguity of the effect allele we decided to drop the BC-NHL (ID 5) and NBS (ID 106) data sets.Reported effect alleles were compatible Figure 3 (Continued) frequency conflicts with the GWAS catalogue are declared if the effect allele frequency is not greater (or less) than 0.5 in both data sets.The level of conflict is further labelled as high if the minor allele frequency is 0.4 in both data sets, and as moderate if not.Effect allele frequency conflicts are also defined as high if the effect allele frequency differs by >10 points between the test and reference data sets.EAF, effect allele frequency; GWAS, genome-wide association study; MAF, minor allele frequency with reported cancer hits in the GWAS catalogue for other cancer data sets (Supplementary Figure S13, available as Supplementary data at IJE online).We identified three data sets in which reported allele frequency was inconsistent with allele frequency in reference data sets (Figures 4 and Supplementary Figures S11  and S12, available as Supplementary data at IJE online).This included GliomaScan (ID 967) in which the allele frequency was inversely correlated with the allele frequency across all SNPs in ancestry matched reference data sets, indicating that the reported effect allele frequency corresponded to the non-effect allele (Figure 4).In the UCSF_AGS/SFAGS (ID 133) and TNC (ID ¼ 132), there were two groups of SNPs showing positive or inverse correlations with the allele frequency in ancestry matched data sets (Supplementary Figures S11 and S12, available as Supplementary data at IJE online; cancer study abbreviations explained in Supplementary Table S6, available as Supplementary data at IJE online), indicating that the reported effect allele frequency actually corresponds to the minor allele frequency and that the minor allele was not consistently the effect allele.For these data sets, we decided Further details of the studies, such as PubMed identifiers and explanations of study abbreviations, can be found in Supplementary Tables S5 and S6 (available as Supplementary data at IJE online).Figure 4. Quality control report for genetic summary data from a genome-wide association of glioma in the GliomaScan data set (ID 967).Allele frequencies are expected to be <0.50.A high allele frequency conflict is defined as an allele frequency of >0.58 in the test data set (GliomaScan in this example) or if the allele frequency differs by >10 points between the test and reference data sets.Moderate allele frequency conflicts are allele frequencies of >0.50 but 0.58.Effect size conflicts are defined as different directions of effect, represented by signed z-scores, between the test data set (GliomaScan in this example) and the GWAS catalogue.The level of conflict is further labelled as 'high' if the P-value for the association is <0.0001 in both the GWAS catalogue and the test data set, and as 'moderate' if not.Effect allele frequency conflicts with the GWAS catalogue are declared if the between studies (Supplementary Figures S25 and S26, available as Supplementary data at IJE online).In 53 data sets (from 25 studies) in which information on imputation quality was available, there were 46 534 SNPs with imputation quality scores of <0.8, including 1119 in the fatty acid SNP set.
Discrepancies between the reported and expected effect sizes in seven data sets from OCAC and PRACTICAL were mainly attributable to a small number of SNPs with unusually large log odds ratios (>1 or < -1) (Supplementary Figure S18, available as Supplementary data at IJE online; cancer study abbreviations explained in Supplementary Table S6, available as Supplementary data at IJE online).The number of SNPs across all data sets with log odds ratios of >1 or < -1 was 368, including five SNPs in the fatty acid SNP set.Additional potential problems in reported effect sizes were identified in two data sets from the UCSF_AGS/SFAGS and ILCCO studies, where the correlation was <0.99 between z p -scores (z-scores inferred from P-values) and z b -scores (z-scores inferred from reported effect sizes and standard errors) (Supplementary Figures S11  and S27, available as Supplementary data at IJE online).In one data set, this was due to three SNPs with very large effect sizes (z > 99) but with P-values very close to 1 (>0.9).The second data set showed a very irregular non-linear relationship between the two sets of z-scores (Supplementary Figure S27, available as Supplementary data at IJE online).This data set was excluded.Correlations between the z band z p -scores were >0.99 across other cancer data sets (Supplementary Figure S28, available as Supplementary data at IJE online).

Final collection of cancer summary-data sets
Application of the QC pipeline to cancer data sets led to the exclusion of 3 data sets and 1 study, leaving 160 data sets from 51 studies (Supplementary Figure S29, available as Supplementary data at IJE online).[101][102] The median number of cases per study was 2442 (minimum ¼ 95; maximum ¼ 122 977) (Supplementary Figure S31, available as Supplementary Figure 4. (Continued) effect allele frequency is not greater (or less) than 0.5 in both data sets.The level of conflict is further labelled as high if the minor allele frequency is 0.4 in both data sets, and as moderate if not.Effect allele frequency conflicts are also defined as high if the effect allele frequency differs by >10 points between the test and reference data sets.CHARGE, Cohorts for Heart and Aging Research in Genomic Epidemiology Consortium; EAF, effect allele frequency; GWAS, genome-wide association study; MAF, minor allele frequency data at IJE online).Fifteen studies reported >10 000 cases, 25 studies reported 1000-10 000 cases and 11 studies reported <1000 cases.

Discussion
Our pipeline flagged analytical issues, metadata and summary-data errors in 23 studies (2 fatty acid GWAS and 21 cancer GWAS), including errors in 7 studies with the potential to introduce substantial bias into downstream MR analyses.These included a large number of false positive genetic associations for fatty acids and incorrect specification of the effect allele and effect allele frequency columns.Other more minor issues included inconsistent effect size scales amongst cancer studies, incorrect assumptions about sample sizes across SNPs and outlier SNPs with unusually large effect sizes.

Effect allele metadata errors
Of the issues identified, incorrect specification of the effect allele column is the most serious, as it will lead to inferences of causal effect in the wrong direction 103,104 (when the null hypothesis is false) and was flagged in 3 of 54 cancer studies.A related, albeit less serious, error is incorrect specification of the effect allele frequency column, which can cause harmonization problems for palindromic SNPs.Failure to harmonize palindromic SNPs between exposure and outcome studies may lead to increased heterogeneity in MR findings, which could in turn bias results towards the null (assuming the null hypothesis is false and that the palindromic SNPs are valid instruments).A conventional approach for avoiding these metadata errors is to compare allele frequency between the GWAS of interest and an external reference data set 12 or to confirm the effect allele through correspondence with study authors (especially when these are ambiguously labelled) or through consultation of readme files.Despite performing the latter checks, five cancer studies were still affected by effect allele metadata errors.One of the metadata errors was introduced by the FAMRC data analyst whereas others were potentially due to human error by data providers.Our approach of comparing summary associations statistics for known 'top hits' between the GWAS of interest and the GWAS catalogue offers an additional safeguard against such errors.

False positive GWAS hits
False positive genetic associations for fatty acids were identified in two of six fatty acid consortia.Failure to account for false positive hits could lead to the inclusion of genetic variants in MR analyses that are not truly associated with the exposure [a violation of instrumental variable assumptions (see Box 1)], which could have the effect of biasing MR findings towards the null (assuming the null hypothesis is false).The false positives arose because we designed our instruments using the full summary association statistics, downloaded from the consortium website or obtained via correspondence, that had not gone through post-GWAS filtering procedures (e.g.exclusion of low frequency or low imputation quality variants).This instrument design strategy is probably more susceptible to inclusion of false positive genetic associations compared with using the manually curated findings described in a GWAS publication.The latter are subject to relatively rigorous reporting standards, whereas there is little consensus on the format that GWAS results should take when posted to study-specific websites.Online platforms and databases that aggregate full summary association statistics from different studies may also be susceptible to this kind of error.
It is important to consider the impact of sample size when interpreting the presence of GWAS hits in the test data set that are absent from the GWAS catalogue.For example, if the GWAS being investigated is unpublished and is larger than any previously published study, we can reasonably expect a number of genetic associations to be identified that are absent from the GWAS catalogue but are nevertheless true novel hits.When the GWAS being investigated is smaller than any previously published study, one should be more sceptical of any GWAS hits that are previously unreported.

Inconsistent effect size scales
We also found that cancer studies did not consistently express effect sizes as log odds ratios, with a substantial proportion of cancer analyses within UK Biobank expressing effect sizes as absolute changes in disease risk.The cancer analyses in question employed BOLT-LMM-a linear mixed model that allows the inclusion of related individuals, is more powerful and efficient than conventional regression procedures 41 and is a widely used method for analysing binary disease traits in large-scale biobanks. 105n general, failure to account for effect size scale differences will hamper comparison of findings amongst different studies and could lead to the misinterpretation of results.

Summary-data errors
Potential summary-data errors were flagged by mismatches between expected and reported effect sizes.We found that a substantial proportion of the mismatches were attributable to imputed SNPs, SNPs with incorrect sample sizes and SNPs with unusually large effect sizes.The sample size errors were due to the strategy of using the maximum reported sample size to represent sample size across all SNPs.However, not all samples in a GWAS necessarily contribute to the analysis of every SNP, which is particularly common in large meta-analysis consortia with many independent studies.Incomplete sample overlap amongst SNPs within a GWAS could introduce bias into methods that assume a constant sample size, such as summary-data methods that rely on an external LD reference panel to model the correlation structure amongst SNPs in a genetic instrument.In the presence of incomplete sample overlap amongst SNPs, the use of an external LD reference panel could lead to the overestimation of the covariance in SNP effect sizes.For example, in the most extreme case of zero sample overlap, the correlation in effect sizes for two SNPs will be zero even if those two SNPs are in LD. 106

General recommendations
When obtaining summary GWAS data via correspondence with study authors, we recommend that researchers should request access to full GWAS summary data, as this allows a far more comprehensive assessment of summary-data reliability than is possible with only subsets of data.When full access is not possible, researchers should request summary data for SNPs that are established GWAS hits for their outcome of interest (i.e.not just the SNPs being used to instrument the exposure), which can then be used to confirm the identity of the effect allele through comparisons with the GWAS catalogue.In addition, researchers could request summary data corresponding to the SNPs in our 1000 genomes reference set, which contains 2297 SNPs with the same minor allele across all 1000 genomes super populations, and which can be used to identify allele frequency issues.An advantage of using our 1000 genomes reference set is that effect allele frequency conflicts can be identified without knowledge of the ancestral background of the test data set.Alternatively, a similar QC check can be achieved by comparing allele frequencies between the exposure and outcome studies of interest (assuming they are closely matched on ancestry).Where possible, researchers should also confirm the identity of the effect allele metadata through correspondence with the data providers.
We also recommend that researchers confirm with data providers the nature of all post-GWAS filtering procedures that have been applied to the summary data.For example, in our own collaboration, we ask each cancer study to confirm that their summary data have been through the same QC procedures as described in their GWAS publications.Failure to perform this check could lead to the inclusion of large numbers of low-quality and unreliable genetic associations.It is also advisable to confirm effect size scales, to support the correct interpretation of results.These considerations supplement previously developed guidelines for conducting MR studies. 4,107,108ur approach of comparing expected to reported effect sizes, and of comparing summary association statistics to external reference data sets, offers an additional safeguard against the aforementioned errors and analytical issues.A limitation of this approach is that not all flagged data sets will necessarily be problematic because other factors, such as covariate adjustment in the original GWAS or deviations from HWE for reasons other than measurement error, could also cause deviations between expected and reported effect sizes.Therefore, SNPs flagged by this approach may still be suitable for downstream MR analyses.
A limitation of our comparative approach is that it may be less effective when there are zero, or few, known genetic associations for the trait of interest.This could happen, for example, when working with understudied or rare characteristics, for which existing published GWAS may be underpowered.In such a situation, comparisons with genetic associations for closely related traits could still be informative.Alternatively, there are a growing number of online platforms that collate summary data from multiple GWAS, which in principle could also be considered as reference data sets when the trait of interest is absent from the GWAS catalogue.These include OpenGWAS (https:// gwas.mrcieu.ac.uk/),GWAS ATLAS (https://atlas.ctglab.nl/), GWAS Central (https://www.gwascentral.org/),PhenoScanner (http://www.phenoscanner.medschl.cam.ac.uk/) and Global Biobank Engine (https://biobankengine. stanford.edu/).
We manually mapped the text descriptions for each cancer type to the EFO, which could be inefficient when working with hundreds or thousands of traits.A more efficient approach would be to use the EMBL-EBI Zooma (https:// www.ebi.ac.uk/spot/zooma) ontology mapping service, which supports command line access via a REST API.

Two-sample population assumption
One of the key assumptions made in two-sample MR is that the studies used to define the exposure and the outcome come from the same population.The comparison of allele frequencies between test data sets and reference populations can in principle be used to evaluate this assumption.For example, in our own analyses, allele frequencies in the European origin cancer studies and 1000 genomes European super population were consistently strongly correlated (the same applied to the East Asian origin studies and the 1000 genomes East Asian super population), indicating that the reported study ancestries were broadly accurate.However, our QC procedure was not designed to specifically test for ancestral origins and was restricted to SNPs with a narrow allele frequency range.A more efficient approach would be to select SNPs with a much wider range of variation in minor allele frequency than chosen here.The need to assess the 'same population' assumption is becoming more urgent with the growing diversity of GWAS, including a growing number of trans-ethnic and admixed studies.

Conclusion
We have developed a QC pipeline that can be used to flag metadata and summary-data errors and a range of analytical issues in GWAS results, which in turn can be used to enhance the integrity of downstream two-sample MR analyses.We applied the pipeline to the FAMRC, identifying errors with potential to introduce substantial bias in seven studies.After resolving analytical issues and excluding problematic studies, 160 data sets from 51 studies were retained, representing 90 unique cancer types generated in analyses of 566 665 cancer cases and 1 622 374 controls.The methods developed here are available to other researchers via the CheckSumStats R package (https:// github.com/MRCIEU/CheckSumStats).

Figure 2
Figure 2 Recommended procedure to confirm the identity of the effect allele and effect allele frequency columns in the results of a genome-wide association study.EBI, European Bioinformatics Institute; EFO, experimental factor ontology; EAF, effect allele frequency; GWAS, genome-wide association study; MAF, minor allele frequency; NHGRI, National Human Genome Research Institute; SNP, single-nucleotide polymorphism

Table 1
Cancer studies in the Fatty Acids in Cancer Mendelian Randomization Collaboration